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What is Claimed: 

1 1 . A method for accurately estimating a pose of a camera within a 

2 scene using a three dimension model of the scene, comprising the steps of : 

3 (a) generating an initial estimate of the pose; 

4 (b) selecting a set of relevant features of the three 

5 dimensional model based on the initial estimate of the pose; 

6 ( C ) creating a virtual projection of the set of relevant 

7 features responsive to the initial estimate of the pose; 

8 (d) matching a plurality of features of an image received 

9 from the camera to the virtual projection of the set of relevant 

10 features and measuring a plurality of matching errors; and 

11 (e) updating the estimate of the pose to reduce the 

12 plurality of matching errors. 

1 2. The method for claim 1 , further comprising the step of: 

2 (f) repeating to steps (c), (d), and (e) using the updated 

3 estimate of the pose until the plurality of matching errors are less 

4 than predetermined matching criteria. 

1 3. The method of claim 1, further comprising the steps of: 

2 (f) adding the updates to the estimate of the pose made 

3 in step (e) to a total change value of the estimate of the pose; 

4 (g) if the total change value is less than predetermined 

5 pose change criteria, resetting the total change value and repeating 

6 to steps (b), (c), (d), (e) and (f) using the updated estimate of the 

7 pose; and 
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8 (h) repeating to steps (c), (d), (e), (f) and (g) using the 

9 updated estimate of the pose until the plurality of matching errors 
io are less than predetermined matching criteria. 

1 4. The method of claim 1, wherein step (a) includes the step of 

2 comparing an image received from the camera to the three dimensional model of the 

3 scene to generate the estimate of the pose. 

1 5. The method of claim 1, wherein step (a) includes the step of using a 

2 preceding pose estimate of a preceding image received from the camera to generate the 

3 estimate of the pose. 

1 6. The method of claim 1, wherein step (b) includes the steps of: 

2 (bl) Z-buffering the three dimensional 

3 model based on the estimated pose to determine a set of visible 

4 features of the three dimensional model; and 

5 (b2) selecting the set of relevant features 

6 from the set of visible features. 

1 7. The method of claim 1, wherein step (b) includes the steps of: 

2 (bl) creating a set of features of the three 

3 dimensional model including a plurality of edges of at least one 

4 object represented in the three dimensional model, each edge 

5 having a dihedral angle of greater than a predetermined angle; 

6 (bl) selecting the set of relevant features 

7 from the set of features of the three dimensional model. 

1 8. The method of claim 1, wherein step (c) includes the step of 

2 removing a model feature from the set of relevant features if the model feature is less than 

3 a predetermined distance from a remaining relevant feature in the virtual projection. 

l 9. The method of claim 1, further comprising the step of: 
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2 (f) perturbing the estimate of the pose and 

3 repeating to steps (c), (d), and (e) using the perturbed estimate of 

4 the pose until the plurality of matching errors are less than 

5 predetermined matching criteria. 

1 10. The method of claim 1, wherein step (d) includes the steps of: 

2 (dl) computing an oriented energy image of 

3 the image received from the camera; and 

4 (d2) integrating the oriented energy image 

5 along the virtual projection of the set of relevant features and 

6 measuring the plurality of matching errors. 

1 11. The method of claim 1, wherein step (d) includes the steps of: 

2 (dl) computing an oriented energy image of 

3 the image received from the camera; 

4 (d2) generating a set of scaled oriented energy 

5 images; 

6 (d3) selecting a scaled oriented energy image 

7 from the set of scaled oriented energy images; and 

8 (d4) integrating the selected scaled oriented 

9 energy image along the virtual projection of the set of relevant 
10 features and measuring the plurality of matching errors. 

1 12. A method for accurately estimating a pose of an image of a scene 

2 using a three dimension model of the scene, comprising the steps of: 

3 (a) performing a pyramid decomposition of the image to 

4 generate a set of pyramid levels of the image; 

5 (b) selecting one pyramid level from the set of pyramid 

6 levels; 
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7 (c) generating an estimate of the pose using the selected 

8 pyramid level; 

9 (d) selecting a set of relevant features of the three 

10 dimensional model based on the estimate of the pose; 

11 (e) creating a virtual projection of the set of relevant 

12 features responsive to the estimate of the pose; 

13 (f) matching a plurality of features of the selected 

14 pyramid level to the virtual projection of the set of relevant features 

15 and measuring a plurality of matching errors; 

16 (g) updating the estimate of the pose to reduce the 

17 plurality of matching errors; and 

18 (h) repeating to steps (e), (f), and (g) using the updated 

19 estimate of the pose until the plurality of matching errors are less 

20 than predetermined matching criteria which is responsive to the 

21 selected pyramid level. 

1 13. The method of claim 12, further comprising the steps of; 

2 (i) removing the selected pyramid level from the set of 

3 pyramid levels; and 

4 (j) repeating to steps (b), (c), (d), (e), (f), (g), (h) and 

5 (i) until the set of pyramid levels contains no elements; 

6 wherein the selected level in step (b) has a lowest resolution level of the 

7 pyramid levels in the set of pyramid levels; . 

1 14. The method of claim 12, wherein step (e) includes the step of 

2 removing a model feature from the set of relevant features if the model feature is less than 

3 a predetermined distance from another relevant feature in the virtual projection, the 

4 predetermined distance being responsive to the selected pyramid level. 
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1 15 . A method for refining a three dimensional model of a scene using 

2 an image of the scene taken by a camera having an unknown pose, comprising the steps 

3 of: 

4 (a) comparing the image to the three dimension model of 

5 the scene to generate an estimate of the pose; and 

6 (b) updating the three dimensional model of the scene 

7 based on data from the image and the estimate of the pose. 

1 16. The method claim 15, wherein step (b) includes the step of: 

2 (bl) projecting a plurality of color values of the image 

3 onto the three dimension model of the scene to update a texture of a 

4 surface of the three dimensional model. 

1 17. A method for accurately estimating a position of remote vehicle 

2 using a three dimension model and an image from a camera having a known orientation 

3 relative to the remote vehicle, comprising the steps of: 

4 (a) comparing the image to the three dimension model of 

5 the scene to generate an estimate of the pose; 

6 (b) selecting a set of relevant features of the three 

7 dimensional model based on the estimate of the pose; 

8 (c) matching a plurality of features of the image to the 

9 set of relevant features and measuring a plurality of matching 
10 errors; 

n (d) updating the estimate of the pose based on the 

12 plurality of matching errors; and 

13 (e) determining the position of the remote vehicle based 

14 on the estimate of the pose and the orientation of the camera. 

1 18. The method of claim 17, further comprising the step of: 



SAR-13825 



-28- 



2 (f) updating the three dimensional model based on data 

3 from the image and the updated estimate of the pose. 

1 19. A method for refining a three dimension model of a scene 

2 containing an object using a plurality of images of the scene, each image including the 

3 object, comprising the steps of: 

4 (a) comparing a first image of the plurality of images to 

5 the three dimension model of the scene to generate an estimate of a 

6 first viewpoint corresponding to the first image; 

7 (b) comparing a second image of the plurality of images 

8 to the three dimension model of the scene to generate an estimate of 

9 a second viewpoint corresponding to the second image; 

10 (c) selecting a first set of relevant features of the three 

11 dimensional model based on the first viewpoint; 

12 (d) matching a plurality of first features of the first 

13 image to the first set of relevant features and measuring a plurality 

14 of first matching errors; 

15 (e) selecting a second set of relevant features of the three 

16 dimensional model based on the second viewpoint; 

17 (f) matching a plurality of second features of the second 

18 image to the second set of relevant features and measuring a 

19 plurality of second matching errors; and 

20 (g) updating a position estimate of the object within the 

21 three dimensional model of the scene based on the plurality of first 

22 matching errors and the plurality of second matching errors. 

1 20. The method of claim 19, further including the steps of: 

2 comparing the first image and the second image to generate relative pose 

3 constraints; and 
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4 updating the first viewpoint and the second viewpoint based on the relative 

5 pose constraints; 

1 21 . The method of claim 20, wherein step of comparing the first image 

2 to the second image includes the steps of: 

3 matching the plurality of first features of the first image to the plurality of 

4 second features of the second feature to create a set of matched features; 

5 calculating optical flow for each matched feature; and 

6 generating the relative pose constraints responsive to the calculated optical 

7 flow. 

1 22. The method of claim 20, wherein the step of comparing the first 

2 image to the second image includes the steps of; 

3 generating a plurality of parallax measurements between the plurality of 

4 first features and the plurality of second features; and 

5 generating the relative pose constraints responsive to the plurality of 

6 parallax measurements. 

1 23 . The method of claim 20, wherein step of comparing the first image 

2 to the second image includes the step of generating the relative pose constraints uses an 

3 epipolar geometry of the first image and the second image. 

1 24. A method for refining a three dimension model of a scene 

2 containing an object using a plurality of images of the scene, each image including the 

3 object, comprising the steps of: 

4 (a) selecting a subset of images from the plurality of 

5 images of the scene, the subset of frames containing at least two of 

6 the images; 

7 (b) determining a plurality of approximate relative 

8 viewpoints of the subset of images; 
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9 (c) comparing each image in the subset of images to the 

10 three dimensional model to generate a subset of estimated viewpoints 

11 corresponding to the subset of images, the subset of estimated 

12 viewpoints constrained by the plurality of approximate relative 

13 viewpoints; 

14 (d) selecting a set of relevant features of the three 

15 dimensional model corresponding to each estimated viewpoint; 

16 (e) matching a plurality of features of the each image in 

17 the subset of images to the corresponding set of relevant features 

18 and measuring a plurality of matching errors; and 

19 (f) updating a position estimate of the object within the 

20 three dimensional model of the scene based on the plurality of 

21 matching errors. 

1 25. The method of claim 24, further comprising the step of: 

2 (g) repeating to steps (c), (d), (e), and (f) using the 

3 updated position estimate of the object until the 

4 plurality of matching errors are less than 

5 predetermined matching criteria. 

1 26. The method of claim 24, wherein the subset of images is selected to 

2 have viewpoints within a predetermined range. 

1 27. A method for creating a hybrid three dimension model of a scene 

2 using a plurality of images of the scene, comprising the steps of: 

3 (a) creating a polyhedral model the scene including at 

4 least one polygonal surface; 

5 (b) determining a first set of images containing at least a 

6 section of a first polygonal surface; and 
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7 ( c ) comparing a plurality of images selected from the 

8 first set of images to generate a local surface shape map 

9 corresponding to the first polygonal surface. 

1 28. The method of claim 27, wherein step (c) includes the steps of; 

2 (cl) calculating optical flow in regions corresponding to the first 

3 polygonal surface of pairs of the first set of images; and 

4 (c2) generating the local surface shape map responsive to the 

5 calculated optical flow. 

1 29. The method of claim 27, wherein step (c) includes the steps of; 

2 (cl) calculating parallax in regions corresponding to the first 

3 polygonal surface of pairs of the first set of images; and 

4 (c2) generating the local surface shape map responsive to the 

5 calculated parallax. 

1 30. The method of claim 27, further comprising the steps of: 

2 (d) separating the local surface shape map of the first 

3 polygonal surface into a plurality of portions of not greater than the 

4 predetermined size; 

5 (e) determining a first subset of images from the first set 

6 of images, each image of the first subset of images containing a first 

7 portion of the first polygonal surface; 

8 (f) selecting at least one selected image of the first 

9 subset of images; 

10 (g) projecting a corresponding section of the at least one 

11 selected image onto the first portion of the local surface shape map 

12 of the first polygonal surface of the hybrid three dimension model 

13 as a local color map; and 
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14 (h) repeating steps (e), (f), and (g) for each remaining 

15 portion of the local surface shape map of the first polygonal 

16 surface. 

1 31 . The method of claim 30, further comprising the step of: 

2 (i) blending the plurality of local color maps 

3 corresponding to the plurality of portions of the local surface shape 

4 map of the first polygonal surface. 

1 32. A method for creating a textured three dimension model of a scene 

2 using a plurality of images of the scene, comprising the steps of: 

3 (a) creating a polyhedral model the scene including at 

4 least one polygonal surface; 

5 (b) identifying at least one portion of the one polygonal 

6 surface; 

7 (c) determining a first subset of images containing the at 

8 least one portion of the one polygonal surface; 

9 (d) selecting at least one selected image of the first 
io subset of images; 

n (e) projecting a corresponding section of each of the at 

12 least one selected image onto the at least one portion of the one 

13 polygonal surface of the textured three dimension model as a local 

14 color map; 

15 (f) repeating steps (c), (d), and (e) for each remaining 

16 portion of the one polygonal surface of the polyhedral model 

l 33. The method of claim 32, wherein step (d) includes the steps of: 
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2 (dl) determining a resolution level for the at 

3 least one portion of the one polygonal surface of each image of 

4 the first subset of images; and 

5 (d2) selecting a selected image of the first 

6 subset of images corresponding to a largest resolution level for 

7 the at least one portion of the one polygonal surface. 

1 34. The method of claim 32, wherein: 

2 the at least one selected image contains at least two selected images; and 

3 step (e) includes the step of at least one of combining and selecting at least 

4 one color value of the at least one corresponding section of the at least two 

5 selected images to create the local color map of the at least one portion of the one 

6 polygonal surface. 

1 35. The method of claim 32, wherein the at least one portion includes a 

2 plurality of portions and the method further comprises the step of: 

3 (g) blending the plurality of local color maps 

4 corresponding to the plurality of portions of the one 

5 polygonal surface. 

1 36. The method of claim 35, wherein step (g) includes the step of: 

2 (gl) creating a plurality of resolution levels of 

3 the plurality of local color maps; and 

4 (g2) blending the plurality of local color maps 

5 over each resolution level. 

1 37. A method for creating a dynamic sequence of virtual images of a 

2 scene using a dynamically updated three dimension model of the scene, comprising the 

3 steps of: 
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4 (a) updating the three dimension model using a video 

5 sequence of images of the scene including the steps of; 

6 (al) determining a present viewpoint of a 

7 present image of the video sequence of images; 

8 (a2) determining a relevant portion of the three 

9 dimensional model corresponding to the present image of the 
10 video sequence of images; 

n (a3) updating the relevant portion of the three 

12 dimensional model by projecting the present image onto the 

13 relevant portion of the three dimension model; 

14 (b) selecting a first virtual viewpoint of a first virtual 

15 image of the dynamic sequence of virtual images; 

16 (c) creating the first virtual image by projecting the 

17 dynamic three dimensional model onto the first virtual viewpoint; 

18 and 

19 (d) repeating steps (b) and (c) for each remaining virtual 

20 image of the dynamic sequence of virtual images. 

1 38. The method of claim 37, wherein step (a) further includes the step 

2 of; 

3 (a4) repeating step (a) using a remaining image 

4 of the sequence of images as the present 

5 image; 

1 39. The method of claim 37, wherein the selection of the virtual 

2 viewpoint in step (b) is responsive to at least one of; 



an external signal; 

a predetermined trajectory; 
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5 a motion of a selected pattern of the dynamic three-dimensional 

6 model; 

7 a motion of a selected object of the dynamic three-dimensional 

8 model; 

9 a motion of a selected pattern of a plurality of images of the 
10 sequence of video images; and 

n a motion of a selected object of a plurality of images of the 

12 sequence of video images. 

1 40. The method of claim 37, wherein the determination of the present 

2 viewpoint of the video sequence of images in step (a) is responsive to at least one of; 

3 an external signal; 

4 a predetermined trajectory; 

5 a motion of a selected pattern of the dynamic three-dimensional 

6 model; 

7 a motion of a selected object of the dynamic three-dimensional 

8 model; 

9 a motion of a selected pattern of a plurality of images of the 

10 sequence of video images; 

11 a motion of a selected object of a plurality of images of the 

12 sequence of video images; and 

13 the virtual viewpoint selected in step (b). 

1 41. A computer readable medium adapted to instruct a general purpose 

2 computer to update a three dimensional model of a scene using the three dimensional 

3 model of the scene, an image received from a camera having an unknown pose, the 

4 method comprising the steps of: 
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5 (a) generating an estimate of the pose; 

6 (b) selecting a set of relevant features of the three 

7 dimensional model based on the estimate of the pose; 

8 (c) creating a virtual projection of the set of relevant 

9 features responsive to the estimate of the pose; 

10 (d) matching a plurality of features of an image received 

11 from the camera to the virtual projection of the set of relevant 

12 features and measuring a plurality of matching errors; 

13 (e) updating the estimate of the pose to reduce the 

14 plurality of matching errors; and 

15 (f) updating the three dimensional model of the scene 

16 based on data from the image and the estimate of the pose. 

1 42. An automatic three-dimensional model updating apparatus for 

2 accurately estimating a point of view of an image of a scene, relative to a three- 

3 dimensional model of the scene, and updating the three-dimensional model comprising: 

4 (a) estimating means for providing an estimate of the 

5 point of view of the image; 

6 (b) relevant feature selecting means for selecting a set of 

7 relevant features of the three dimensional model based on the 

8 estimate of the point of view; 

9 (c) virtual projection means for creating a virtual 

10 projection of the set of relevant features responsive to the estimate 

11 of the point of view; 

12 (d) matching means for matching a plurality of features 

13 of the image to the virtual projection of the set of relevant features; 
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14 (e) measurement means for measuring a plurality of 

15 matching errors; 

16 (f) point of view refinement means for updating the 

17 estimate of the point of view to reduce the plurality of matching 

18 errors; and 

19 (g) model refinement means, responsive to the estimated 

20 point of view and to the image, for updating the three-dimensional 

21 model. 



